A model and measure of agreement for population-based studies

Agreement between physicians in their classification of items such as mammograms for the presence of disease is an important tool in assessing the reliability of a diagnostic procedure, and the modeling of agreement data is a popular topic in the biomedical and social sciences. Interest often lies in assessing agreement in the underlying diagnostic procedure and making inferences for the populations of raters and items typically involved in the rating process. However, the majority of methods currently available are limited to inference for the specific groups of raters and items selected for study, and most do not apply when many raters are involved. In this paper we describe the use of generalized linear mixed models with crossed random effects to model agreement between many raters and items over the long-run for classifications made on a binary scale. These models flexibly allow for missing and unbalanced data, many raters and items, the inclusion of covariates that may influence the agreement process and most importantly, provides inference regarding the underlying diagnostic process and the populations of the typical raters and items involved in such classifications. To provide an overall measure of agreement we propose a summary model-based statistic which is easily interpretable in a manner similar to Cohen’s kappa statistic, while avoiding some of the biases that arise in Cohen’s kappa usage. The proposed agreement measure can also be used to describe agreement between subgroups of raters and items by utilizing available covariate information. Simulation studies demonstrate that the proposed approach provides unbiased chance-corrected estimates of agreement. The methods are applied to an agreement dataset involving the classification of mammograms for the presence/absence of breast cancer (Beam 2003).